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Why run NGINX in a cluster? 


æ HSS 


e Fault Tolerance 
e Scalability 
e Topology Requirements 
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Four important concerns 


How do | manage How do I monitor 
e configuration? e thecluster? 
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How do | distribute » How do | share 
e traffic? e runtime state? 


Four important concerns 


Configuration Management 
How do I manage 
e configuration? You have a huge choice: 


Roll your own, nginx -t and 
SIGHUP 

Configuration Management 
tooling: Ansible, Chef, Puppet 
Immutable Containers 
NGINX Controller 


Four important concerns 


Cluster Monitoring 
How do I monitor 
e the cluster? An entire industry exists to help you 
solve this problem: 


FA DATADOG 


A Prometheus Nagios’ 


fD 19 Grafana 


dynatrace 


librato © influxdb 


©) APPDYNAMICS 


NGINX Controller N) 


Four important concerns 


How do | distribute P How do I share 
e traffic? e runtime state? 


Distribute Traffic - Keepalived 
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Keepalived 


Keepalived daemon runs on each NGINX 
instance 


e Manages one or more VIPs 
e Performs health checks against its peers 
e Shares state using VRRP/multicast 


e Raises each VIP on a working NGINX 


instance (primary) 
e Re-raises the VIP if the primary fails 
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Distribute Traffic - Keepalived 


vrrp_instance VI_1 { 


interface 
priority 
virtual router _id 
advert_int 
accept 
garp_ master refresh 5 
garp master refresh_repeat 1 
unicast_src_ip 192.168.100.100 
unicast_peer { 
VIP 


192.168.100.101 
} 


virtual_ipaddress { 
192.168.100.150 


} 


track_script { 
chk_nginx_service 
chk_manual_failover 
} 


Keepalived notify "/usr/libexec/keepalived/nginx-ha-notify" 


Distribute Traffic - Keepalived 


e Strengths: 
o Simple, mature, well-understood, easy to extend 
o Can be customized to work in, for example, cloud environments 
o NGINX Plus: fully-supported solution with setup script and support 


e Weaknesses: 
o Multi-active and multi-passive configurations need care 
o Multi-active does not share traffic; requires load balancer or RRDNS 


Distribute Traffic - IPVS 


Operates in a L4 DSR fashion 


e All NGINX instances raise the VIP (local 
scope) 

e One NGINX instance (the Primary) 
advertises the VIP and uses ipvs to load- 
balance packets across the cluster 


e Keepalived performs health checks and 
failover 


Keepalived 
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Distribute Traffic - IPVS 


Keepalived 


Raise the traffic IP with local scope 
ip addr add 192.168.56.40/32 scope host 
Ñ 


dev lo 


Only IPs with global scope respond to ARP requests 


sysctl net.ipv4.conf.all.arp_ignore=3 
sysctl net.ipv4.conf.all.arp_announce=2 


Use keepalived to set up the rest: 
/usr/local/sbin/keepalived 


Keepalived understands DR mode 
# virtual server config 
virtual_server 192.168.56.40 80 { 
lb_algo rr 
lb_kind DR 
protocol TCP 


Distribute Traffic - IPVS 


e Strengths: 
o Distributes TCP connections across cluster effectively 
o Managed by keepalived 
o Options to tune lb_algo, weights 


e Weaknesses: 


o Primary failure disrupts all established connections (state sharing is 
possible) 

o Adds additional hop -> perceived latency issue (unlikely to be 
significant) 

o Asymmetric traffic distribution (Primary takes 2x Ingress bandwidth) 

o Complex to debug in the event of issues 


Distribute Traffic - Multicast 


Ingress 


A Egress 
————A 


Keepalived / Pacemaker 


Traffic distributed to all instances 


e All NGINX instances raise the VIP with a 
common multicast MAC address 

e Allinstances receive ingress traffic 

e Each instance has a kernel filter that 
silently drops all but 1/N of the traffic 

e Every connection handled by precisely one 
NGINX instance 


e Keepalived / pacemaker can perform 
health checks and failover 


Distribute Traffic - Multicast 


Ingress 


Keepalived / Pacemaker 


Join the Multicast group: 
smcroute -j enpOs8 239.104.4.4 


Raise our VIP IP Address: 
ip addr add dev enpOs8 192.168.20.40 


Hard-code the ARP response using the mcast MAC 

arptables -A OUTPUT -s 192.168.20.40 \ 
--h-length 6 -j mangle \ 
--mangle-mac-s 01:00:5e:68:04:04 


Configure the ClusterlP module to accept our share 
iptables -A INPUT -d 192.168.20.40 \ 
-i enpOs8 -p tcp -j CLUSTERIP --new \ 
--hashmode sourceip-sourceport \ 
--clustermac 01:00:5e:68:04:04 \ 
--total-nodes 4 --local-node 4 


Distribute Traffic - Multicast 


e Strengths: 
o Distributes TCP connections across cluster effectively 
o No single primary makes configuration simpler 


e Weaknesses: 
o Only supported in environments that support multicast 
o Basic hash-based load balancing 
o Failover is an exercise for the reader: 
echo +4 > /proc/net/ipt CLUSTERIP/192.168.20.40 
Pacemaker and Keepalived both provide options 


Distribute Traffic - ECMP 


Traffic distributed to all instances 


e Requires front-end ECMP/BGP router 
ECMP router e Consistent-Hashing can be used to 
distribute traffic efficiently 


e Ideal if you have the support of your 
infrastructure team to configure and 
manage 
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Four important concerns 


How do I share 
e runtime state? 


NGINX — 


Thank you 


vl@nginx.com 
owen @nginx.com 


